Lipreadability of a synthetic talking face in normal hearing and hearing-impaired listeners

The Synface project is developing a synthetic talking face to aid the hearing-impaired in telephone conversation. This report investigates the gain in intelligibility from the synthetic talking head when controlled by hand-annotated speech in both 12 normal hearing (NH) and 13 hearing-impaired (HI) listeners (average hearing loss 86 dB). For NH listeners, audio from everyday sentences was degraded to simulate the information losses that arise in severe-to-profound hearing impairment. For the HI group, audio was filtered to simulate telephone speech. Auditory signals were presented alone, with the synthetic face, and with a video of the original talker. Purely auditory intelligibility was low for the NH group. With the addition of the synthetic face, average intelligibility increased by 22%. The HI group had a large variation in intelligibility in the purely auditory condition. They showed a 22% improvement with the addition of the synthetic face. For both groups, intelligibility with the synthetic face was significantly lower than with the natural face. However, the improvement with the synthetic face is sufficient to be useful in everyday communication. Questionnaire responses from the HI group indicated strong interest in the Synface system.